ENH: Add use_nullable_dtypes in csv internals #48403
Conversation
phofl
added
IO CSV
| @@ -15,6 +15,13 @@ import warnings | |||
|
|
|||
| from pandas.util._exceptions import find_stack_level | |||
|
|
|||
| from pandas import StringDtype | |||
| from pandas.core.arrays import ( | |||
There was a problem hiding this comment.
is there a viable way to do this outside of the cython code?
There was a problem hiding this comment.
A possible option might be to change _maybe_upcast to optionally return a values array + mask array, and then do the actual construction in python?
Although since _maybe_upcast is only being called in this file (and thus from cython), that won't help. Unless if we would propagate such a (values, mask) tuple (instead of ArrayLike) through the different calls and return that from the TextReader.read method.
There was a problem hiding this comment.
yah not worth contorting ourselves to avoid non-cython imports. just if there's a convenient alternative
There was a problem hiding this comment.
Could be done yes, but would make logic more complex, which is something we should avoid here I think
There was a problem hiding this comment.
yah, never mind then. thanks for taking a look
…e_dtypes_in_maybe_upcast
…e_dtypes_in_maybe_upcast
…e_dtypes_in_maybe_upcast
|
Thanks @phofl |
* ENH: Add use_nullable_dtypes in csv internals * Add tests * Fix mypy * Add comment * Add pyarrow test * Fix float32 * Fix float32 * Add contextmanager
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This adds the required casting logic for extension array dtypes. Want to get this done as a first step before adding a public option to read_csv. The unit tests can be used as a guideline for the required behaviour.
This always casts to ea dtypes, even when no missing values are present. From a users perspective this makes the most sense imo. You don't want to do a follow up operation and end up with float in a pipeline, when you set nullable dtypes (reindexing, enlargement in indexing ops, ...). That said, if there are concerns about this, we could also do something like
use_nullable_dtypes=False | always | when necessarycc @jorisvandenbossche